Modeling Intensive Longitudinal Behavioral Data with Dynamic Structural Equation Models

Mannheim - 21 September, 2023

Jean-Paul Snijder

University of Heidelberg

1 Before we start

  • Formality:
    • Feel free to call me Jean-Paul or JP
    • Feel free to ask questions during the talk (with a handraise)
    • Feel free to ask me to slow down
    • Feel free to ask me to speed up
  • Work in Progress
    • Started in January/February
    • Suggestions welcome
    • Share ideas

2 The Past: Individual Differences in Cognitive Control

The path I took - Graduate School

Masters and Ph.D. in applied cognitive psychology

  • Claremont Graduate University
  • Andrew Conway
    • Working Memory Span tasks
    • Intelligence
      • Process Overlap Theory

The path I took - Early career

Masters

  • Thesis: The Effect of Working Memory Training on Cognitive Control and Reading Comprehension
  • Modeling Working Memory

Ph.D.

  • Simulating intelligence data for POT
  • Psychometrics
  • Dissertation: A Hierarchical Bayesian Approach to Estimating Individual Differences Reliability in Classic Cognitive Control Tasks

Summary of my dissertation

“The basic ability in cognitive control is maintaining a goal, and its goal-relevant information, in the face of distraction”

Also called:

  • attentional control
  • executive functioning
  • executive attention

Friedman and Miyake: Unity and Diversity Model

  • Response Inhibition
  • Working Memory Updating
  • Taskset Shifting

Cognitive control tasks

Stroop:

  • Color (ink) naming task
    • congruent (matching color-ink/word)
    • incongruent (mismatched color-ink/word)

  • Stroop effect:

    • \(RT\)incongruent - \(RT\)congruent

    • \(ACC\)incongruent - \(ACC\)congruent

Assumed underlying mechanism:

  • pre-potent response inhibition
    • automated reading

Experimental effect

Cognitive control effects:

  • Reliable
    • Robust
    • “Everybody Stroops” - Jeff Rouder

“Every healthy person Stroops”

  • Used to measure:
    • A.D.D
    • Schizophrenia
    • Frontal lobe damage

Individual Differences

Stroop and other cognitive control tasks have inconsistencies in validity

  • convergent
  • predictive

Rey-Mermet et al., 2018

Why?

Low below between-subject variance

experimental tasks are created to

  • maximize within-subject variance
    • e.g.; difference between congruent & incongruent
  • minimize between-subject variance
    • is treated as measurement noise

Additional reasons

furthermore, researchers proposed

  • reliability (Hedge et al., 2018)
    • reliability is a bottleneck for correlations
  • difference scores (Von Bastian et al., 2020)
    • notoriously problematic for psychometrics
    • example with Stroop
  • aggregate statistics (Rouder & Haaf, 2019)
    • mean instead of MLM
  • not a psychometric construct at all! (Rey-Mermet et al., 2018)
    • task-specific variance, not a single construct: cognitive control

My dissertation

  • Difference scores (Von Bastian et al., 2020)
    • latent modeling
  • Aggregate statistics (Rouder & Haaf, 2019)
    • MLM (RTs as lognormal)
  • Reliability (Hedge et al., 2018)
    • poor validity stems from poor reliability?

Plus: Theory-based task manipulations based on the Dual Mechanisms of Cognitive Control framework

My dissertation

Dual Mechanisms of Cognitive Control (Braver, 2007; 2012)

  • Proactive

Proactive control refers to a sustained and anticipatory mode of control that allows individuals to actively and optimally configure processing resources before the onset of task demands

  • Reactive

Involves a ‘wait-and-see’ mode of control that is triggered by stimuli, and relies upon retrieval of task goals and the rapid mobilization of processing resources after the onset of a cognitively demanding event

My dissertation

4 Tasks:

  • Stroop, AX-CPT, Task-Switching, & Sternberg WM
  • Baseline, Proactive, Reactive
  • Test + Retest: 6 weeks
  • ~ 130 subjects

Analysis first phase:

Classical Frequentist Approach

Analysis second phase:

Hierarchical Bayesian Modeling

Frequentist Reliability

Frequentist Validity

Bayesian Reliability

Bayesian Reliability

Reliability Bottlenecks Validity?

if reliability indeed bottlenecks the between-task correlations

and we improved reliability with the HB model

then what do we expect for the convergent validity (between-task correlations)?

Bayesian Phase Validity

Wrap up

meta-analysis, von Bastian et al., 2020

Wrap up

  • Methodological exhaustion?
    • Hierarchical Bayesian models
    • Drift diffusion models
    • SEM/FA
    • Previously unseen combinations of methods.

“Neither measurement error nor speed-accuracy trade-offs explain the difficulty of establishing attentional control as a psychometric construct: Evidence from a latent-variable analysis using diffusion modeling”

An old idea

  • Not a psychometric construct at all! (Rey-Mermet et al., 2018)
    • task-specific variance, not a single construct: cognitive control
  • Still curious what else could be done

An old idea

  • An old project, 2nd semester masters

RTs and WMC, McVay & Kane, 2011

An old idea

RTs and WMC, McVay & Kane, 2011

An old idea

RTs and WMC, McVay & Kane, 2011

What are some analyses that you use?

  • mean (sum in case of survey)
  • location/scale (e.g., \(\mathcal{N}\)(\(\mu\), \(\sigma ^2\)))

  • Noise around the mean instead of the mean?

    • Paying attention example

  • Intra-Individual Variability (IIV)

  • Variability is shaped by randomly occurring, inordinately slow trials

    • Aristodemou et al., 2022; Kofler et al., 2013; Geurts et al., 2008

Intra-Individual Variability

Snijder et al., 2021

Intra-Individual Variability

Snijder et al., 2021

Measuring IIV

  • \(\mathbf{\sigma^2}\) is crude
    • correlation between RT Mean and RT SD (Wagenmakers & Brown, 2007)
  • Coefficient of Variation (CV)
    • \(\text{CV} = \frac{\sigma_i}{y_i}\) (SD/Mean)
  • Intra-individual Standard Deviation (ISD)
    • \({\text{ISD}} = \sqrt{\sum \frac{ (y_{i,t} - y_i)^2} {T_i} }\)
  • Ex-Gaussian \(\tau\) parameter
  • However…

Measuring IIV: trend?

  • Variability is shaped by randomly occurring, inordinately slow trials
  • Wang et al., 2012:
    • systematic (non-random) variability: trends (i.e., learning/improvements)
    • CV and ISD:
      • Subject who steadily improves/declines, mistaken for high variability
    • Ex-Gaussian
      • Sensitive to trend effects

???

A novel way of measuring IIV?

a plane? a bird? no, it’s …

???

A novel way of measuring IIV?

a plane? a bird? no, it’s DSEM

Short pause

  • Next: an introduction DSEM (15 minutes)
  • Then: finish with a variant of DSEM
    • ties everything together

3 The Present: DSEM Tutorial

Dynamic Structural Equation Modeling (DSEM)

  • Technological advancements are increasing availability of Intensive Longitudinal Data (ILD) from:
    • Experience Sampling Methods (ESM, EMA, AA)
    • Electro-EncephaloGram (EEG)
    • Wearables
  • ILD are densely spaced repeated measures data collected from large samples
  • Need for models that allow to examine dynamic changes over time
  • Computational models are developed and adapted to match this growing demand

Dynamic Structural Equation Modeling

Combines: 1

  • Time-series modeling
    • allows lagged relationships
  • Multilevel modeling
    • allows modeling of nested data structures
  • Structural equation modeling
    • allows latent variable/path analysis

DSEM in Mplus

Pros

  • Widely used
  • online user and program support
  • Considered user-friendly
  • Low computational time
    • Gibbs sampler with conjugate priors (Normal \(\Leftrightarrow\) Inverse Wishart)

Cons

  • Not fully customizable
  • currently doesn’t support some model extensions and specifications
  • limited prior options and access to sampler settings
    • i.e., no LKJ distribution
  • Limited options for missing data
  • License costs money

Stan

Pros

  • Free
  • Fully customizable
  • Open Code & Reproducible Science
  • Online community support
  • Hamiltonian Monte Carlo
    • Efficient general-purpose MCMC sampler

Cons

  • Programming can pose a barrier
    • Fully code-based
    • No GUI
  • Higher computational time
    • but reasonable (minutes to hours)
    • not optimized for a specific model family

DSEM software alternatives

  • dlsem in R:
    • Uses frequentist inference
  • ctsem in R:
    • Slow for full Bayesian estimation
    • Oriented towards continuous time systems
      • but discrete can be used
    • Less user-friendly
    • No latent classes and limited non-continuous measurement models
  • JAGS

Our project

  • Stan tutorial using DSEM framework as example
    1. Introducing DSEM
    2. Improving the accessibility to Stan
  • 6 model archetypes1
    1. Bivariate, Single Case
    2. Bivariate, Multilevel
    3. Model 2 + predictor variable
    4. Model 2 + latent variable
    5. Model 3 + outcome variable
    6. Model 4 + mediation

Our project

Taken from 1 slightly altered for fit

Our project

  • Stan tutorial using DSEM framework as example
    1. Introducing DSEM
    2. Improving the accessibility to Stan
  • 6 model archetypes1
    1. Bivariate, Single Case
    2. Bivariate, Multilevel
    3. Model 2 + predictor variable
    4. Model 2 + latent variable
    5. Model 3 + outcome variable
    6. Model 4 + mediation
  • For each archetype in Stan:
    1. Simple: tutorial model
    2. Reparam: reparameterized model
    3. Full: missing data model

DSEM

M2: Two Variable, Multilevel Model

  • Model 2: two variables + multilevel
    • Stress
    • Sleep

M2: Two Variable, Multilevel Model

  • Model 2: two variables + multilevel
    • Stress
    • Sleep

  • Within- & between-person decomposition
    • Between: time-insensitive mean of subject
    • Within: time-sensitive deviation from that mean
  • Allows for specifying time-dynamics in within-person model

M2: Within-person Model I

  • The decomposed within-person variables are the start of the within-person model \(\rightarrow\)

M2: Within-person Model II

Relationships & Parameters:

  • Regression:
    • \(\beta\)YX = Stresst regressed on Sleept

M2: Within-person Model II

Relationships & Parameters:

  • Regression:
    • \(\beta\)YX = Stresst regressed on Sleept
  • Time Dynamic Regressions:
    • \(\Phi\)X,i = auto-regressive parameter Stress
    • \(\Phi\)Y,i = auto-regressive parameter Sleep
      • Stressi,t-1(w) and Sleepi,t-1(w) are lag(1) variables
      • E.g., if t = observation 9 \(\Rightarrow\) t-1 = observation 8 …

M2: Within-person Model II

Relationships & Parameters:

  • Regression:
    • \(\beta\)YX = Stresst regressed on Sleept
  • Time Dynamic Regressions:
    • \(\Phi\)X,i = auto-regressive parameter Stress
    • \(\Phi\)Y,i = auto-regressive parameter Sleep
    • \(\Phi\)XY,i = cross-regressive parameter Sleepi,t onto Stressi,t-1
  • Residual variances:
    • \(\Psi\)2X,i and \(\Psi\)2Y,i

M2: Between-person Model

Stan

Within-level model I

\[\begin{align} Stress_{i,t}^{(w)} &= \mathcal{N}([\Phi_{X,i}] [Stress_{i,t-1}^{(w)}] + [\beta_{YX,i}] [Sleep_{i,t}^{(w)}], \Psi_{X,i}^2) \\ Sleep_{i,t}^{(w)} &= \mathcal{N}([\Phi_{Y,i}] [Sleep_{i,t-1}^{(w)}] + [\Phi_{XY,i}] [Stress_{i,t-1}^{(w)}], \Psi_{Y,i}^2) \\ \end{align}\]

Within-level model II

Stress_t ~ normal(phi_X * Stress_t_1 + beta_YX * Sleep_t, psi_X);
Sleep_t ~ normal(phi_Y * Sleep_t_1 + phi_XY * Stress_t_1, psi_Y);

\[\begin{align} Stress_{i,t}^{(w)} &= \mathcal{N}([\Phi_{X,i}] [Stress_{i,t-1}^{(w)}] + [\beta_{YX,i}] [Sleep_{i,t}^{(w)}], \Psi_{X,i}^2) \\ Sleep_{i,t}^{(w)} &= \mathcal{N}([\Phi_{Y,i}] [Sleep_{i,t-1}^{(w)}] + [\Phi_{XY,i}] [Stress_{i,t-1}^{(w)}], \Psi_{Y,i}^2) \\ \end{align}\]

Between-level model I

  • Using latent means and random intercepts/effects

\[\begin{align} X_i^{(b)} &= \gamma_1 + u_{i1} \\ Y_i^{(b)} &= \gamma_2 + u_{i2} \\ \Phi_{Xi} &= \gamma_3 + u_{i3} \\ \Phi_{Yi} &= \gamma_4 + u_{i4} \\ \Phi_{XYi} &= \gamma_5 + u_{i5} \\ \beta_{YXi} &= \gamma_6 + u_{i6} \\ \log\Psi_{Xi}^2 &= \gamma_7 + u_{i7} \\ \log\Psi_{Yi}^2 &= \gamma_8 + u_{i8} \\ \end{align}\]

\[\begin{align} \boldsymbol{u}\sim\text{MVNormal}(\boldsymbol{0}, \boldsymbol{\Omega}) \end{align}\]

Between-level model II

  • Using latent means and random intercepts/effects


real mu_X = gamma[1] + u[i,1];
real mu_Y = gamma[2] + u[i,2];

real phi_X = gamma[3] + u[i,3];
real phi_Y = gamma[4] + u[i,4];
real phi_XY = gamma[5] + u[i,5];
real beta_YX = gamma[6] + u[i,6];

real psi_X = sqrt(exp(gamma[7] + u[i,7]));
real psi_Y = sqrt(exp(gamma[8] + u[i,8]));

u[i] ~ multi_normal(rep_vector(0, 8), Omega);

\[\begin{align} X_i^{(b)} &= \gamma_1 + u_{i1} \\ Y_i^{(b)} &= \gamma_2 + u_{i2} \\ \Phi_{Xi} &= \gamma_3 + u_{i3} \\ \Phi_{Yi} &= \gamma_4 + u_{i4} \\ \Phi_{XYi} &= \gamma_5 + u_{i5} \\ \beta_{YXi} &= \gamma_6 + u_{i6} \\ \log\Psi_{Xi}^2 &= \gamma_7 + u_{i7} \\ \log\Psi_{Yi}^2 &= \gamma_8 + u_{i8} \\ \end{align}\]

\[\begin{align} \boldsymbol{u}\sim\text{MVNormal}(\boldsymbol{0}, \boldsymbol{\Omega}) \end{align}\]

Optimization: Reparameterization

  • Improves convergence, can speed up sampling
  • Classical example: \(y \sim \mathcal{N}(\mu, \sigma^2) \Leftrightarrow y = \mu + \sigma\cdot\tilde y\text{ with }\tilde y\sim\mathcal{N}(0, 1)\)

Handling missing data

  • Missing data is unknown
  • Parameters are unknown

\(\rightarrow\) treat missing data like parameters

  • Preserves uncertainty (unlike mean imputation etc.)

Simulation

Results with simulated data

  • Model 2
  • 100 subjects
  • 100 observations
  • relevant parameter ranges for sleep and stress
  • for missing data model: 5% missingness
  • no model misspecification
  • Sampler:
    • 500 warmup/3500 sampling iterations
    • 4 chains, 16 cores

Model convergence

Model 2, simulated data convergence.

Parameter recovery

Model 2, simulated data. Errorbars: 95% CI.

Current and Future Work

  • current: Simulations
    • relevant parameter ranges
    • prior calibration
  • current: Finish writing the paper
  • near: Standardized estimates
  • near: Model implementation for cognitive behavioral tasks
  • far: R Package with Stan as back-end

4 The Future: Linear Growth AR(1) Model

LGAR

Linear Growth AR(1) Model:

  • estimate of IIV (\(\psi_i^2\))
  • accounts for trend (\(\beta\))
    • Wang et al., 2012
  • accounts for auto-regression (\(\phi\))

Plot

Stan

normal_lpdf(w_Xt| alpha + beta * t + phi * w_Xt_1, psi);

Stan

real alpha = gamma[1] + u[i,1]; 
real beta = gamma[2] + u[i,2];
real phi = gamma[3] + u[i,3];
real psi = sqrt(exp(gamma[4] + u[i,4]));

for (t in 2:N_obs) {
  // calculate innovations
  real w_Xt = X_imp[i,t] - alpha; 
  real w_Xt_1 = X_imp[i,t-1] - alpha; 
  
  ll += normal_lpdf(w_Xt| alpha + beta * t + phi * w_Xt_1, psi);
}

Prelimenary results

Prelimenary results

Thanks to

Valentin Pratz - simulations and stan models

Thank you

Github repo with presentation + reproducable model 2 example

Hamaker, E. L., Asparouhov, T., & Muthén, B. (2023). Dynamic Structural Equation Modeling as a Combination of Time Series Modeling, Multilevel Modeling, and Structural Equation Modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (2nd ed., pp. 576–597). New York: Guilford Press.